Apparatus and method for implementing a multi-level memory hierarchy over common memory channels

ABSTRACT

A system and method are described for integrating a memory and storage hierarchy including a non-volatile memory tier within a computer system. In one embodiment, PCMS memory devices are used as one tier in the hierarchy, sometimes referred to as “far memory.” Higher performance memory devices such as DRAM placed in front of the far memory and are used to mask some of the performance limitations of the far memory. These higher performance memory devices are referred to as “near memory.”

BACKGROUND

1. Field of the Invention

This invention relates generally to the field of computer systems. Moreparticularly, the invention relates to an apparatus and method forimplementing a multi-level memory hierarchy.

2. Description of the Related Art

A. Current Memory and Storage Configurations

One of the limiting factors for computer innovation today is memory andstorage technology. In conventional computer systems, system memory(also known as main memory, primary memory, executable memory) istypically implemented by dynamic random access memory (DRAM). DRAM-basedmemory consumes power even when no memory reads or writes occur becauseit must constantly recharge internal capacitors. DRAM-based memory isvolatile, which means data stored in DRAM memory is lost once the poweris removed. Conventional computer systems also rely on multiple levelsof caching to improve performance. A cache is a high speed memorypositioned between the processor and system memory to service memoryaccess requests faster than they could be serviced from system memory.Such caches are typically implemented with static random access memory(SRAM). Cache management protocols may be used to ensure that the mostfrequently accessed data and instructions are stored within one of thelevels of cache, thereby reducing the number of memory accesstransactions and improving performance.

With respect to mass storage (also known as secondary storage or diskstorage), conventional mass storage devices typically include magneticmedia (e.g., hard disk drives), optical media (e.g., compact disc (CD)drive, digital versatile disc (DVD), etc.), holographic media, and/ormass-storage flash memory (e.g., solid state drives (SSDs), removableflash drives, etc.). Generally, these storage devices are consideredInput/Output (I/O) devices because they are accessed by the processorthrough various I/O adapters that implement various I/O protocols. TheseI/O adapters and I/O protocols consume a significant amount of power andcan have a significant impact on the die area and the form factor of theplatform. Portable or mobile devices (e.g., laptops, netbooks, tabletcomputers, personal digital assistant (PDAs), portable media players,portable gaming devices, digital cameras, mobile phones, smartphones,feature phones, etc.) that have limited battery life when not connectedto a permanent power supply may include removable mass storage devices(e.g., Embedded Multimedia Card (eMMC), Secure Digital (SD) card) thatare typically coupled to the processor via low-power interconnects andI/O controllers in order to meet active and idle power budgets.

With respect to firmware memory (such as boot memory (also known as BIOSflash)), a conventional computer system typically uses flash memorydevices to store persistent system information that is read often butseldom (or never) written to. For example, the initial instructionsexecuted by a processor to initialize key system components during aboot process (Basic Input and Output System (BIOS) images) are typicallystored in a flash memory device. Flash memory devices that are currentlyavailable in the market generally have limited speed (e.g., 50 MHz).This speed is further reduced by the overhead for read protocols (e.g.,2.5 MHz). In order to speed up the BIOS execution speed, conventionalprocessors generally cache a portion of BIOS code during thePre-Extensible Firmware Interface (PEI) phase of the boot process. Thesize of the processor cache places a restriction on the size of the BIOScode used in the PEI phase (also known as the “PEI BIOS code”).

B. Phase-Change Memory (PCM) and Related Technologies

Phase-change memory (PCM), also sometimes referred to as phase changerandom access memory (PRAM or PCRAM), PCME, Ovonic Unified Memory, orChalcogenide RAM (C-RAM), is a type of non-volatile computer memorywhich exploits the unique behavior of chalcogenide glass. As a result ofheat produced by the passage of an electric current, chalcogenide glasscan be switched between two states: crystalline and amorphous. Recentversions of PCM can achieve two additional distinct states.

PCM provides higher performance than flash because the memory element ofPCM can be switched more quickly, writing (changing individual bits toeither 1 or 0) can be done without the need to first erase an entireblock of cells, and degradation from writes is slower (a PCM device maysurvive approximately 100 million write cycles; PCM degradation is dueto thermal expansion during programming, metal (and other material)migration, and other mechanisms).

BRIEF DESCRIPTION OF THE DRAWINGS

The following description and accompanying drawings are used toillustrate embodiments of the invention. In the drawings:

FIG. 1 illustrates a cache and system memory arrangement according toembodiments of the invention;

FIG. 2 illustrates a memory and storage hierarchy employed inembodiments of the invention;

FIG. 3 illustrates a computer system on which embodiments of theinvention may be implemented;

FIG. 4A illustrates a first system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4B illustrates a second system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4C illustrates a third system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4D illustrates a fourth system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4E illustrate a fifth system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4F illustrate a sixth system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4G illustrates a seventh system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4H illustrates an eight system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4I illustrates a ninth system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4J illustrates a tenth system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4K illustrates an eleventh system architecture which includes PCMaccording to embodiments of the invention;

FIG. 4L illustrates a twelfth system architecture which includes PCMaccording to embodiments of the invention; and

FIG. 4M illustrates a thirteenth system architecture which includes PCMaccording to embodiments of the invention.

FIG. 5A illustrates one embodiment of a system architecture whichincludes a volatile near memory and a non-volatile far memory;

FIG. 5B illustrates one embodiment of a memory side cache (MSC);

FIG. 5C illustrates another embodiment of a memory side cache (MSC)which includes an integrated tag cache and ECC generation/check logic;

FIG. 5D illustrates one embodiment of an exemplary tag cache and ECCgenerator/check unit;

FIG. 5E illustrates one embodiment of a PCM DIMM including a PCMcontroller;

FIG. 6A illustrates MCE controllers and caches dedicated to certainspecified system physical address (SPA) ranges according to oneembodiment of the invention;

FIG. 6B illustrates an exemplary mapping between a system memory map, anear memory address map and a PCM address map according to oneembodiment of the invention;

FIG. 6C illustrates an exemplary mapping between a system physicaladdress (SPA) and a PCM physical device address (PDA) or a near memoryaddress (NMA) according to one embodiment of the invention; and

FIG. 6D illustrates interleaving between memory pages within a systemphysical address (SPA) space and memory channel address (MCA) spaceaccording to one embodiment of the invention.

DETAILED DESCRIPTION

In the following description, numerous specific details such as logicimplementations, opcodes, means to specify operands, resourcepartitioning/sharing/duplication implementations, types andinterrelationships of system components, and logicpartitioning/integration choices are set forth in order to provide amore thorough understanding of the present invention. It will beappreciated, however, by one skilled in the art that the invention maybe practiced without such specific details. In other instances, controlstructures, gate level circuits and full software instruction sequenceshave not been shown in detail in order not to obscure the invention.Those of ordinary skill in the art, with the included descriptions, willbe able to implement appropriate functionality without undueexperimentation.

References in the specification to “one embodiment,” “an embodiment,”“an example embodiment,” etc., indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it is submitted that it is within the knowledge of oneskilled in the art to effect such feature, structure, or characteristicin connection with other embodiments whether or not explicitlydescribed.

In the following description and claims, the terms “coupled” and“connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended as synonyms for each other.“Coupled” is used to indicate that two or more elements, which may ormay not be in direct physical or electrical contact with each other,co-operate or interact with each other. “Connected” is used to indicatethe establishment of communication between two or more elements that arecoupled with each other.

Bracketed text and blocks with dashed borders (e.g., large dashes, smalldashes, dot-dash, dots) are sometimes used herein to illustrate optionaloperations/components that add additional features to embodiments of theinvention. However, such notation should not be taken to mean that theseare the only options or optional operations/components, and/or thatblocks with solid borders are not optional in certain embodiments of theinvention.

INTRODUCTION

Memory capacity and performance requirements continue to increase withan increasing number of processor cores and new usage models such asvirtualization. In addition, memory power and cost have become asignificant component of the overall power and cost, respectively, ofelectronic systems.

Some embodiments of the invention solve the above challenges byintelligently subdividing the performance requirement and the capacityrequirement between memory technologies. The focus of this approach ison providing performance with a relatively small amount of a relativelyhigher-speed memory such as DRAM while implementing the bulk of thesystem memory using significantly cheaper and denser non-volatile randomaccess memory (NVRAM). Embodiments of the invention described belowdefine platform configurations that enable hierarchical memory subsystemorganizations for the use of NVRAM. The use of NVRAM in the memoryhierarchy also enables new usages such as expanded boot space and massstorage implementations, as described in detail below.

FIG. 1 illustrates a cache and system memory arrangement according toembodiments of the invention. Specifically, FIG. 1 shows a memoryhierarchy including a set of internal processor caches 120, “nearmemory” acting as a far memory cache 121, which may include bothinternal cache(s) 106 and external caches 107-109, and “far memory” 122.One particular type of memory which may be used for “far memory” in someembodiments of the invention is non-volatile random access memory(“NVRAM”). As such, an overview of NVRAM is provided below, followed byan overview of far memory and near memory.

A. Non-Volatile Random Access Memory (“NVRAM”)

There are many possible technology choices for NVRAM, including PCM,Phase Change Memory and Switch (PCMS) (the latter being a more specificimplementation of the former), byte-addressable persistent memory(BPRAM), universal memory, Ge2Sb2Te5, programmable metallization cell(PMC), resistive memory (RRAM), RESET (amorphous) cell, SET(crystalline) cell, PCME, Ovshinsky memory, ferroelectric memory (alsoknown as polymer memory and poly(N-vinylcarbazole)), ferromagneticmemory (also known as Spintronics, SPRAM (spin-transfer torque RAM),STRAM (spin tunneling RAM), magnetoresistive memory, magnetic memory,magnetic random access memory (MRAM)), andSemiconductor-oxide-nitride-oxide-semiconductor (SONOS, also known asdielectric memory).

For use in the memory hierarchy described in this application, NVRAM hasthe following characteristics:

(1) It maintains its content even if power is removed, similar to FLASHmemory used in solid state disks (SSD), and different from SRAM and DRAMwhich are volatile;

(2) lower power consumption when idle than volatile memories such asSRAM and DRAM;

(3) random access similar to SRAM and DRAM (also known as randomlyaddressable);

(4) rewritable and erasable at a lower level of granularity (e.g., bytelevel) than FLASH found in SSDs (which can only be rewritten and eraseda “block” at a time—minimally 64 Kbyte in size for NOR FLASH and 16Kbyte for NAND FLASH);

(5) usable as a system memory and allocated all or a portion of thesystem memory address space;

(6) capable of being coupled to the processor over a bus using atransactional protocol (a protocol that supports transaction identifiers(IDs) to distinguish different transactions so that those transactionscan complete out-of-order) and allowing access at a level of granularitysmall enough to support operation of the NVRAM as system memory (e.g.,cache line size such as 64 or 128 byte). For example, the bus may be amemory bus (e.g., a DDR bus such as DDR3, DDR4, etc.) over which is runa transactional protocol as opposed to the non-transactional protocolthat is normally used. As another example, the bus may one over which isnormally run a transactional protocol (a native transactional protocol),such as a PCI express (PCIE) bus, desktop management interface (DMI)bus, or any other type of bus utilizing a transactional protocol and asmall enough transaction payload size (e.g., cache line size such as 64or 128 byte); and

(6) one or more of the following:

a) faster write speed than non-volatile memory/storage technologies suchas FLASH;b) very high read speed (faster than FLASH and near or equivalent toDRAM read speeds);c) directly writable (rather than requiring erasing (overwriting with1s) before writing data like FLASH memory used in SSDs);d) orders of magnitude (e.g., 2 or 3) higher write endurance beforefailure (more than boot ROM and FLASH used in SSDs); and/or

As mentioned above, in contrast to FLASH memory, which must be rewrittenand erased a complete “block” at a time, the level of granularity atwhich NVRAM is accessed in any given implementation may depend on theparticular memory controller and the particular memory bus or other typeof bus to which the NVRAM is coupled. For example, in someimplementations where NVRAM is used as system memory, the NVRAM may beaccessed at the granularity of a cache line (e.g., a 64-byte or 128-Bytecache line), notwithstanding an inherent ability to be accessed at thegranularity of a byte, because cache line is the level at which thememory subsystem accesses memory. Thus, when NVRAM is deployed within amemory subsystem, it may be accessed at the same level of granularity asthe DRAM (e.g., the “near memory”) used in the same memory subsystem.Even so, the level of granularity of access to the NVRAM by the memorycontroller and memory bus or other type of bus is smaller than that ofthe block size used by Flash and the access size of the I/O subsystem'scontroller and bus.

NVRAM may also incorporate wear leveling algorithms to account for thefact that the storage cells at the far memory level begin to wear outafter a number of write accesses, especially where a significant numberof writes may occur such as in a system memory implementation. Sincehigh cycle count blocks are most likely to wear out in this manner, wearleveling spreads writes across the far memory cells by swappingaddresses of high cycle count blocks with low cycle count blocks. Notethat most address swapping is typically transparent to applicationprograms because it is handled by hardware, lower-level software (e.g.,a low level driver or operating system), or a combination of the two.

B. Far Memory

The far memory 122 of some embodiments of the invention is implementedwith NVRAM, but is not necessarily limited to any particular memorytechnology. Far memory 122 is distinguishable from other instruction anddata memory/storage technologies in terms of its characteristics and/orits application in the memory/storage hierarchy. For example, far memory122 is different from:

1) static random access memory (SRAM) which may be used for level 0 andlevel 1 internal processor caches 101 a-b, 102 a-b, 103 a-b, 103 a-b,and 104 a-b dedicated to each of the processor cores 101-104,respectively, and lower level cache (LLC) 105 shared by the processorcores;2) dynamic random access memory (DRAM) configured as a cache 106internal to the processor 100 (e.g., on the same die as the processor100) and/or configured as one or more caches 107-109 external to theprocessor (e.g., in the same or a different package from the processor100); and3) FLASH memory/magnetic disk/optical disc applied as mass storage (notshown); and4) memory such as FLASH memory or other read only memory (ROM) appliedas firmware memory (which can refer to boot ROM, BIOS Flash, and/or TPMFlash).(not shown).

Far memory 122 may be used as instruction and data storage that isdirectly addressable by a processor 100 and is able to sufficiently keeppace with the processor 100 in contrast to FLASH/magnetic disk/opticaldisc applied as mass storage. Moreover, as discussed above and describedin detail below, far memory 122 may be placed on a memory bus and maycommunicate directly with a memory controller that, in turn,communicates directly with the processor 100.

Far memory 122 may be combined with other instruction and data storagetechnologies (e.g., DRAM) to form hybrid memories (also known asCo-locating PCM and DRAM; first level memory and second level memory;FLAM (FLASH and DRAM)). Note that at least some of the abovetechnologies, including PCM/PCMS may be used for mass storage insteadof, or in addition to, system memory, and need not be random accessible,byte addressable or directly addressable by the processor when appliedin this manner.

For convenience of explanation, most of the remainder of the applicationwill refer to “NVRAM” or, more specifically, “PCM,” or “PCMS” as thetechnology selection for the far memory 122. As such, the terms NVRAM,PCM, PCMS, and far memory may be used interchangeably in the followingdiscussion. However it should be realized, as discussed above, thatdifferent technologies may also be utilized for far memory. Also, thatNVRAM is not limited for use as far memory.

C. Near Memory

“Near memory” 121 is an intermediate level of memory configured in frontof a far memory 122 that has lower read/write access latency relative tofar memory and/or more symmetric read/write access latency (i.e., havingread times which are roughly equivalent to write times). In someembodiments, the near memory 121 has significantly lower write latencythan the far memory 122 but similar (e.g., slightly lower or equal) readlatency; for instance the near memory 121 may be a volatile memory suchas volatile random access memory (VRAM) and may comprise a DRAM or otherhigh speed capacitor-based memory. Note, however, that the underlyingprinciples of the invention are not limited to these specific memorytypes. Additionally, the near memory 121 may have a relatively lowerdensity and/or may be more expensive to manufacture than the far memory122.

In one embodiment, near memory 121 is configured between the far memory122 and the internal processor caches 120. In some of the embodimentsdescribed below, near memory 121 is configured as one or morememory-side caches (MSCs) 107-109 to mask the performance and/or usagelimitations of the far memory including, for example, read/write latencylimitations and memory degradation limitations. In theseimplementations, the combination of the MSC 107-109 and far memory 122operates at a performance level which approximates, is equivalent orexceeds a system which uses only DRAM as system memory. As discussed indetail below, although shown as a “cache” in FIG. 1, the near memory 121may include modes in which it performs other roles, either in additionto, or in lieu of, performing the role of a cache.

Near memory 121 can be located on the processor die (as cache(s) 106)and/or located external to the processor die (as caches 107-109) (e.g.,on a separate die located on the CPU package, located outside the CPUpackage with a high bandwidth link to the CPU package, for example, on amemory dual in-line memory module (DIMM), a riser/mezzanine, or acomputer motherboard). The near memory 121 may be coupled in communicatewith the processor 100 using a single or multiple high bandwidth links,such as DDR or other transactional high bandwidth links (as described indetail below).

An Exemplary System Memory Allocation Scheme

FIG. 1 illustrates how various levels of caches 101-109 are configuredwith respect to a system physical address (SPA) space 116-119 inembodiments of the invention. As mentioned, this embodiment comprises aprocessor 100 having one or more cores 101-104, with each core havingits own dedicated upper level cache (L0) 101 a-104 a and mid-level cache(MLC) (L1) cache 101 b-104 b. The processor 100 also includes a sharedLLC 105. The operation of these various cache levels are well understoodand will not be described in detail here.

The caches 107-109 illustrated in FIG. 1 may be dedicated to aparticular system memory address range or a set of non-contiguousaddress ranges. For example, cache 107 is dedicated to acting as an MSCfor system memory address range #1 116 and caches 108 and 109 arededicated to acting as MSCs for non-overlapping portions of systemmemory address ranges #2 117 and #3 118. The latter implementation maybe used for systems in which the SPA space used by the processor 100 isinterleaved into an address space used by the caches 107-109 (e.g., whenconfigured as MSCs). In some embodiments, this latter address space isreferred to as a memory channel address (MCA) space. In one embodiment,the internal caches 101 a-106 perform caching operations for the entireSPA space.

System memory as used herein is memory which is visible to and/ordirectly addressable by software executed on the processor 100; whilethe cache memories 101 a-109 may operate transparently to the softwarein the sense that they do not form a directly-addressable portion of thesystem address space, but the cores may also support execution ofinstructions to allow software to provide some control (configuration,policies, hints, etc.) to some or all of the cache(s). The subdivisionof system memory into regions 116-119 may be performed manually as partof a system configuration process (e.g., by a system designer) and/ormay be performed automatically by software.

In one embodiment, the system memory regions 116-119 are implementedusing far memory (e.g., PCM) and, in some embodiments, near memoryconfigured as system memory. System memory address range #4 representsan address range which is implemented using a higher speed memory suchas DRAM which may be a near memory configured in a system memory mode(as opposed to a caching mode).

FIG. 2 illustrates a memory/storage hierarchy 140 and differentconfigurable modes of operation for near memory 144 and NVRAM accordingto embodiments of the invention. The memory/storage hierarchy 140 hasmultiple levels including (1) a cache level 150 which may includeprocessor caches 150A (e.g., caches 101A-105 in FIG. 1) and optionallynear memory as cache for far memory 150B (in certain modes of operationas described herein), (2) a system memory level 151 which may includefar memory 151B (e.g., NVRAM such as PCM) when near memory is present(or just NVRAM as system memory 174 when near memory is not present),and optionally near memory operating as system memory 151A (in certainmodes of operation as described herein), (3) a mass storage level 152which may include a flash/magnetic/optical mass storage 152B and/orNVRAM mass storage 152A (e.g., a portion of the NVRAM 142); and (4) afirmware memory level 153 that may include BIOS flash 170 and/or BIOSNVRAM 172 and optionally trusted platform module (TPM) NVRAM 173.

As indicated, near memory 144 may be implemented to operate in a varietyof different modes including: a first mode in which it operates as acache for far memory (near memory as cache for FM 150B); a second modein which it operates as system memory 151A and occupies a portion of theSPA space (sometimes referred to as near memory “direct access” mode);and one or more additional modes of operation such as a scratchpadmemory 192 or as a write buffer 193. In some embodiments of theinvention, the near memory is partitionable, where each partition mayconcurrently operate in a different one of the supported modes; anddifferent embodiments may support configuration of the partitions (e.g.,sizes, modes) by hardware (e.g., fuses, pins), firmware, and/or software(e.g., through a set of programmable range registers within the MSCcontroller 124 within which, for example, may be stored different binarycodes to identify each mode and partition).

System address space A 190 in FIG. 2 is used to illustrate operationwhen near memory is configured as a MSC for far memory 150B. In thisconfiguration, system address space A 190 represents the entire systemaddress space (and system address space B 191 does not exist).Alternatively, system address space B 191 is used to show animplementation when all or a portion of near memory is assigned aportion of the system address space. In this embodiment, system addressspace B 191 represents the range of the system address space assigned tothe near memory 151A and system address space A 190 represents the rangeof the system address space assigned to NVRAM 174.

In addition, when acting as a cache for far memory 150B, the near memory144 may operate in various sub-modes under the control of the MSCcontroller 124. In each of these modes, the near memory address space(NMA) is transparent to software in the sense that the near memory doesnot form a directly-addressable portion of the system address space.These modes include but are not limited to the following:

(1) Write-Back Caching Mode:

In this mode, all or portions of the near memory acting as a FM cache150B is used as a cache for the NVRAM far memory (FM) 151B. While inwrite-back mode, every write operation is directed initially to the nearmemory as cache for FM 150B (assuming that the cache line to which thewrite is directed is present in the cache). A corresponding writeoperation is performed to update the NVRAM FM 151B only when the cacheline within the near memory as cache for FM 150B is to be replaced byanother cache line (in contrast to write-through mode described below inwhich each write operation is immediately propagated to the NVRAM FM151B).

(2) Near Memory Bypass Mode:

In this mode all reads and writes bypass the NM acting as a FM cache150B and go directly to the NVRAM FM 151B. Such a mode may be used, forexample, when an application is not cache friendly or requires data tobe committed to persistence at the granularity of a cache line. In oneembodiment, the caching performed by the processor caches 150A and theNM acting as a FM cache 150B operate independently of one another.Consequently, data may be cached in the NM acting as a FM cache 150Bwhich is not cached in the processor caches 150A (and which, in somecases, may not be permitted to be cached in the processor caches 150A)and vice versa. Thus, certain data which may be designated as“uncacheable” in the processor caches may be cached within the NM actingas a FM cache 150B.

(3) Near Memory Read-Cache Write Bypass Mode:

This is a variation of the above mode where read caching of thepersistent data from NVRAM FM 151B is allowed (i.e., the persistent datais cached in the near memory as cache for far memory 150B for read-onlyoperations). This is useful when most of the persistent data is“Read-Only” and the application usage is cache-friendly.

(4) Near Memory Read-Cache Write-Through Mode:

This is a variation of the near memory read-cache write bypass mode,where in addition to read caching, write-hits are also cached. Everywrite to the near memory as cache for FM 150B causes a write to the FM151B. Thus, due to the write-through nature of the cache, cache-linepersistence is still guaranteed.

When acting in near memory direct access mode, all or portions of thenear memory as system memory 151A are directly visible to software andform part of the SPA space. Such memory may be completely under softwarecontrol. Such a scheme may create a non-uniform memory address (NUMA)memory domain for software where it gets higher performance from nearmemory 144 relative to NVRAM system memory 174. By way of example, andnot limitation, such a usage may be employed for certain highperformance computing (HPC) and graphics applications which require veryfast access to certain data structures.

In an alternate embodiment, the near memory direct access mode isimplemented by “pinning” certain cache lines in near memory (i.e., cachelines which have data that is also concurrently stored in NVRAM 142).Such pinning may be done effectively in larger, multi-way,set-associative caches.

FIG. 2 also illustrates that a portion of the NVRAM 142 may be used asfirmware memory. For example, the BIOS NVRAM 172 portion may be used tostore BIOS images (instead of or in addition to storing the BIOSinformation in BIOS flash 170). The BIOS NVRAM portion 172 may be aportion of the SPA space and is directly addressable by softwareexecuted on the processor cores 101-104, whereas the BIOS flash 170 isaddressable through the I/O subsystem 115. As another example, a trustedplatform module (TPM) NVRAM 173 portion may be used to protect sensitivesystem information (e.g., encryption keys).

Thus, as indicated, the NVRAM 142 may be implemented to operate in avariety of different modes, including as far memory 151B (e.g., whennear memory 144 is present/operating, whether the near memory is actingas a cache for the FM via a MSC control 124 or not (accessed directlyafter cache(s) 101A-105 and without MSC control 124)); just NVRAM systemmemory 174 (not as far memory because there is no near memorypresent/operating; and accessed without MSC control 124); NVRAM massstorage 152A; BIOS NVRAM 172; and TPM NVRAM 173. While differentembodiments may specify the NVRAM modes in different ways, FIG. 3describes the use of a decode table 333.

FIG. 3 illustrates an exemplary computer system 300 on which embodimentsof the invention may be implemented. The computer system 300 includes aprocessor 310 and memory/storage subsystem 380 with a NVRAM 142 used forboth system memory, mass storage, and optionally firmware memory. In oneembodiment, the NVRAM 142 comprises the entire system memory and storagehierarchy used by computer system 300 for storing data, instructions,states, and other persistent and non-persistent information. Aspreviously discussed, NVRAM 142 can be configured to implement the rolesin a typical memory and storage hierarchy of system memory, massstorage, and firmware memory, TPM memory, and the like. In theembodiment of FIG. 3, NVRAM 142 is partitioned into FM 151B, NVRAM massstorage 152A, BIOS NVRAM 173, and TMP NVRAM 173. Storage hierarchieswith different roles are also contemplated and the application of NVRAM142 is not limited to the roles described above.

By way of example, operation while the near memory as cache for FM 150Bis in the write-back caching is described. In one embodiment, while thenear memory as cache for FM 150B is in the write-back caching modementioned above, a read operation will first arrive at the MSCcontroller 124 which will perform a look-up to determine if therequested data is present in the near memory acting as a cache for FM150B (e.g., utilizing a tag cache 342). If present, it will return thedata to the requesting CPU, core 101-104 or I/O device through I/Osubsystem 115. If the data is not present, the MSC controller 124 willsend the request along with the system memory address to an NVRAMcontroller 332. The NVRAM controller 332 will use the decode table 333to translate the system memory address to an NVRAM physical deviceaddress (PDA) and direct the read operation to this region of the farmemory 151B. In one embodiment, the decode table 333 includes an addressindirection table (AIT) component which the NVRAM controller 332 uses totranslate between system memory addresses and NVRAM PDAs. In oneembodiment, the AIT is updated as part of the wear leveling algorithmimplemented to distribute memory access operations and thereby reducewear on the NVRAM FM 151B. Alternatively, the AIT may be a separatetable stored within the NVRAM controller 332.

Upon receiving the requested data from the NVRAM FM 151B, the NVRAMcontroller 332 will return the requested data to the MSC controller 124which will store the data in the MSC near memory acting as an FM cache150B and also send the data to the requesting processor core 101-104, orI/O Device through I/O subsystem 115. Subsequent requests for this datamay be serviced directly from the near memory acting as a FM cache 150Buntil it is replaced by some other NVRAM FM data.

As mentioned, in one embodiment, a memory write operation also firstgoes to the MSC controller 124 which writes it into the MSC near memoryacting as a FM cache 150B. In write-back caching mode, the data may notbe sent directly to the NVRAM FM 151B when a write operation isreceived. For example, the data may be sent to the NVRAM FM 151B onlywhen the location in the MSC near memory acting as a FM cache 150B inwhich the data is stored must be re-used for storing data for adifferent system memory address. When this happens, the MSC controller124 notices that the data is not current in NVRAM FM 151B and will thusretrieve it from near memory acting as a FM cache 150B and send it tothe NVRAM controller 332. The NVRAM controller 332 looks up the PDA forthe system memory address and then writes the data to the NVRAM FM 151B.

In FIG. 3, the NVRAM controller 332 is shown connected to the FM 151B,NVRAM mass storage 152A, and BIOS NVRAM 172 using three separate lines.This does not necessarily mean, however, that there are three separatephysical buses or communication channels connecting the NVRAM controller332 to these portions of the NVRAM 142. Rather, in some embodiments, acommon memory bus or other type of bus (such as those described belowwith respect to FIGS. 4A-M) is used to communicatively couple the NVRAMcontroller 332 to the FM 151B, NVRAM mass storage 152A, and BIOS NVRAM172. For example, in one embodiment, the three lines in FIG. 3 representa bus, such as a memory bus (e.g., a DDR3, DDR4, etc, bus), over whichthe NVRAM controller 332 implements a transactional protocol tocommunicate with the NVRAM 142. The NVRAM controller 332 may alsocommunicate with the NVRAM 142 over a bus supporting a nativetransactional protocol such as a PCI express bus, desktop managementinterface (DMI) bus, or any other type of bus utilizing a transactionalprotocol and a small enough transaction payload size (e.g., cache linesize such as 64 or 128 byte).

In one embodiment, computer system 300 includes integrated memorycontroller (IMC) 331 which performs the central memory access controlfor processor 310, which is coupled to: 1) a memory-side cache (MSC)controller 124 to control access to near memory (NM) acting as a farmemory cache 150B; and 2) a NVRAM controller 332 to control access toNVRAM 142. Although illustrated as separate units in FIG. 3, the MSCcontroller 124 and NVRAM controller 332 may logically form part of theIMC 331.

In the illustrated embodiment, the MSC controller 124 includes a set ofrange registers 336 which specify the mode of operation in use for theNM acting as a far memory cache 150B (e.g., write-back caching mode,near memory bypass mode, etc, described above). In the illustratedembodiment, DRAM 144 is used as the memory technology for the NM actingas cache for far memory 150B. In response to a memory access request,the MSC controller 124 may determine (depending on the mode of operationspecified in the range registers 336) whether the request can beserviced from the NM acting as cache for FM 150B or whether the requestmust be sent to the NVRAM controller 332, which may then service therequest from the far memory (FM) portion 151B of the NVRAM 142.

In an embodiment where NVRAM 142 is implemented with PCMS, NVRAMcontroller 332 is a PCMS controller that performs access with protocolsconsistent with the PCMS technology. As previously discussed, the PCMSmemory is inherently capable of being accessed at the granularity of abyte. Nonetheless, the NVRAM controller 332 may access a PCMS-based farmemory 151B at a lower level of granularity such as a cache line (e.g.,a 64-bit or 128-bit cache line) or any other level of granularityconsistent with the memory subsystem. The underlying principles of theinvention are not limited to any particular level of granularity foraccessing a PCMS-based far memory 151B. In general, however, whenPCMS-based far memory 151B is used to form part of the system addressspace, the level of granularity will be higher than that traditionallyused for other non-volatile storage technologies such as FLASH, whichcan only perform rewrite and erase operations at the level of a “block”(minimally 64 Kbyte in size for NOR FLASH and 16 Kbyte for NAND FLASH).

In the illustrated embodiment, NVRAM controller 332 can readconfiguration data to establish the previously described modes, sizes,etc. for the NVRAM 142 from decode table 333, or alternatively, can relyon the decoding results passed from IMC 331 and I/O subsystem 315. Forexample, at either manufacturing time or in the field, computer system300 can program decode table 333 to mark different regions of NVRAM 142as system memory, mass storage exposed via SATA interfaces, mass storageexposed via USB Bulk Only Transport (BOT) interfaces, encrypted storagethat supports TPM storage, among others. The means by which access issteered to different partitions of NVRAM device 142 is via a decodelogic. For example, in one embodiment, the address range of eachpartition is defined in the decode table 333. In one embodiment, whenIMC 331 receives an access request, the target address of the request isdecoded to reveal whether the request is directed toward memory, NVRAMmass storage, or I/O. If it is a memory request, IMC 331 and/or the MSCcontroller 124 further determines from the target address whether therequest is directed to NM as cache for FM 150B or to FM 151B. For FM151B access, the request is forwarded to NVRAM controller 332. IMC 331passes the request to the I/O subsystem 115 if this request is directedto I/O (e.g., non-storage and storage I/O devices). I/O subsystem 115further decodes the address to determine whether the address points toNVRAM mass storage 152A, BIOS NVRAM 172, or other non-storage or storageI/O devices. If this address points to NVRAM mass storage 152A or BIOSNVRAM 172, I/O subsystem 115 forwards the request to NVRAM controller332. If this address points to TMP NVRAM 173, I/O subsystem 115 passesthe request to TPM 334 to perform secured access.

In one embodiment, each request forwarded to NVRAM controller 332 isaccompanied with an attribute (also known as a “transaction type”) toindicate the type of access. In one embodiment, NVRAM controller 332 mayemulate the access protocol for the requested access type, such that therest of the platform remains unaware of the multiple roles performed byNVRAM 142 in the memory and storage hierarchy. In alternativeembodiments, NVRAM controller 332 may perform memory access to NVRAM 142regardless of which transaction type it is. It is understood that thedecode path can be different from what is described above. For example,IMC 331 may decode the target address of an access request and determinewhether it is directed to NVRAM 142. If it is directed to NVRAM 142, IMC331 generates an attribute according to decode table 333. Based on theattribute, IMC 331 then forwards the request to appropriate downstreamlogic (e.g., NVRAM controller 332 and I/O subsystem 315) to perform therequested data access. In yet another embodiment, NVRAM controller 332may decode the target address if the corresponding attribute is notpassed on from the upstream logic (e.g., IMC 331 and I/O subsystem 315).Other decode paths may also be implemented.

The presence of a new memory architecture such as described hereinprovides for a wealth of new possibilities. Although discussed at muchgreater length further below, some of these possibilities are quicklyhighlighted immediately below.

According to one possible implementation, NVRAM 142 acts as a totalreplacement or supplement for traditional DRAM technology in systemmemory. In one embodiment, NVRAM 142 represents the introduction of asecond-level system memory (e.g., the system memory may be viewed ashaving a first level system memory comprising near memory as cache 150B(part of the DRAM device 340) and a second level system memorycomprising far memory (FM) 151B (part of the NVRAM 142).

According to some embodiments, NVRAM 142 acts as a total replacement orsupplement for the flash/magnetic/optical mass storage 152B. Aspreviously described, in some embodiments, even though the NVRAM 152A iscapable of byte-level addressability, NVRAM controller 332 may stillaccess NVRAM mass storage 152A in blocks of multiple bytes, depending onthe implementation (e.g., 64 Kbytes, 128 Kbytes, etc.). The specificmanner in which data is accessed from NVRAM mass storage 152A by NVRAMcontroller 332 may be transparent to software executed by the processor310. For example, even through NVRAM mass storage 152A may be accesseddifferently from Flash/magnetic/optical mass storage 152A, the operatingsystem may still view NVRAM mass storage 152A as a standard mass storagedevice (e.g., a serial ATA hard drive or other standard form of massstorage device).

In an embodiment where NVRAM mass storage 152A acts as a totalreplacement for the flash/magnetic/optical mass storage 152B, it is notnecessary to use storage drivers for block-addressable storage access.The removal of storage driver overhead from storage access can increaseaccess speed and save power. In alternative embodiments where it isdesired that NVRAM mass storage 152A appears to the OS and/orapplications as block-accessible and indistinguishable fromflash/magnetic/optical mass storage 152B, emulated storage drivers canbe used to expose block-accessible interfaces (e.g., Universal SerialBus (USB) Bulk-Only Transfer (BOT), 1.0; Serial Advanced TechnologyAttachment (SATA), 3.0; and the like) to the software for accessingNVRAM mass storage 152A.

In one embodiment, NVRAM 142 acts as a total replacement or supplementfor firmware memory such as BIOS flash 362 and TPM flash 372(illustrated with dotted lines in FIG. 3 to indicate that they areoptional). For example, the NVRAM 142 may include a BIOS NVRAM 172portion to supplement or replace the BIOS flash 362 and may include aTPM NVRAM 173 portion to supplement or replace the TPM flash 372.Firmware memory can also store system persistent states used by a TPM334 to protect sensitive system information (e.g., encryption keys). Inone embodiment, the use of NVRAM 142 for firmware memory removes theneed for third party flash parts to store code and data that arecritical to the system operations.

Continuing then with a discussion of the system of FIG. 3, in someembodiments, the architecture of computer system 100 may includemultiple processors, although a single processor 310 is illustrated inFIG. 3 for simplicity. Processor 310 may be any type of data processorincluding a general purpose or special purpose central processing unit(CPU), an application-specific integrated circuit (ASIC) or a digitalsignal processor (DSP). For example, processor 310 may be ageneral-purpose processor, such as a Core™ i3, i5, i7, 2 Duo and Quad,Xeon™, or Itanium™ processor, all of which are available from IntelCorporation, of Santa Clara, Calif. Alternatively, processor 310 may befrom another company, such as ARM Holdings, Ltd, of Sunnyvale, Calif.,MIPS Technologies of Sunnyvale, Calif., etc. Processor 310 may be aspecial-purpose processor, such as, for example, a network orcommunication processor, compression engine, graphics processor,co-processor, embedded processor, or the like. Processor 310 may beimplemented on one or more chips included within one or more packages.Processor 310 may be a part of and/or may be implemented on one or moresubstrates using any of a number of process technologies, such as, forexample, BiCMOS, CMOS, or NMOS. In the embodiment shown in FIG. 3,processor 310 has a system-on-a-chip (SOC) configuration.

In one embodiment, the processor 310 includes an integrated graphicsunit 311 which includes logic for executing graphics commands such as 3Dor 2D graphics commands. While the embodiments of the invention are notlimited to any particular integrated graphics unit 311, in oneembodiment, the graphics unit 311 is capable of executing industrystandard graphics commands such as those specified by the Open GL and/orDirect X application programming interfaces (APIs) (e.g., OpenGL 4.1 andDirect X 11).

The processor 310 may also include one or more cores 101-104, although asingle core is illustrated in FIG. 3, again, for the sake of clarity. Inmany embodiments, the core(s) 101-104 includes internal functionalblocks such as one or more execution units, retirement units, a set ofgeneral purpose and specific registers, etc. If the core(s) aremulti-threaded or hyper-threaded, then each hardware thread may beconsidered as a “logical” core as well. The cores 101-104 may behomogenous or heterogeneous in terms of architecture and/or instructionset. For example, some of the cores may be in order while others areout-of-order. As another example, two or more of the cores may becapable of executing the same instruction set, while others may becapable of executing only a subset of that instruction set or adifferent instruction set.

The processor 310 may also include one or more caches, such as cache 313which may be implemented as a SRAM and/or a DRAM. In many embodimentsthat are not shown, additional caches other than cache 313 areimplemented so that multiple levels of cache exist between the executionunits in the core(s) 101-104 and memory devices 150B and 151B. Forexample, the set of shared cache units may include an upper-level cache,such as a level 1 (L1) cache, mid-level caches, such as level 2 (L2),level 3 (L3), level 4 (L4), or other levels of cache, an (LLC), and/ordifferent combinations thereof. In different embodiments, cache 313 maybe apportioned in different ways and may be one of many different sizesin different embodiments. For example, cache 313 may be an 8 megabyte(MB) cache, a 16 MB cache, etc. Additionally, in different embodimentsthe cache may be a direct mapped cache, a fully associative cache, amulti-way set-associative cache, or a cache with another type ofmapping. In other embodiments that include multiple cores, cache 313 mayinclude one large portion shared among all cores or may be divided intoseveral separately functional slices (e.g., one slice for each core).Cache 313 may also include one portion shared among all cores andseveral other portions that are separate functional slices per core.

The processor 310 may also include a home agent 314 which includes thosecomponents coordinating and operating core(s) 101-104. The home agentunit 314 may include, for example, a power control unit (PCU) and adisplay unit. The PCU may be or include logic and components needed forregulating the power state of the core(s) 101-104 and the integratedgraphics unit 311. The display unit is for driving one or moreexternally connected displays.

As mentioned, in some embodiments, processor 310 includes an integratedmemory controller (IMC) 331, near memory cache (MSC) controller, andNVRAM controller 332 all of which can be on the same chip as processor310, or on a separate chip and/or package connected to processor 310.DRAM device 144 may be on the same chip or a different chip as the IMC331 and MSC controller 124; thus, one chip may have processor 310 andDRAM device 144; one chip may have the processor 310 and another theDRAM device 144 and (these chips may be in the same or differentpackages); one chip may have the core(s) 101-104 and another the IMC331, MSC controller 124 and DRAM 144 (these chips may be in the same ordifferent packages); one chip may have the core(s) 101-104, another theIMC 331 and MSC controller 124, and another the DRAM 144 (these chipsmay be in the same or different packages); etc.

In some embodiments, processor 310 includes an I/O subsystem 115 coupledto IMC 331. I/O subsystem 115 enables communication between processor310 and the following serial or parallel I/O devices: one or morenetworks 336 (such as a Local Area Network, Wide Area Network or theInternet), storage I/O device (such as flash/magnetic/optical massstorage 152B, BIOS flash 362, TPM flash 372) and one or more non-storageI/O devices 337 (such as display, keyboard, speaker, and the like). I/Osubsystem 115 may include a platform controller hub (PCH) (not shown)that further includes several I/O adapters 338 and other I/O circuitryto provide access to the storage and non-storage I/O devices andnetworks. To accomplish this, I/O subsystem 115 may have at least oneintegrated I/O adapter 338 for each I/O protocol utilized. I/O subsystem115 can be on the same chip as processor 310, or on a separate chipand/or package connected to processor 310.

I/O adapters 338 translate a host communication protocol utilized withinthe processor 310 to a protocol compatible with particular I/O devices.For flash/magnetic/optical mass storage 152B, some of the protocols thatI/O adapters 338 may translate include Peripheral Component Interconnect(PCI)-Express (PCI-E), 3.0; USB, 3.0; SATA, 3.0; Small Computer SystemInterface (SCSI), Ultra-640; and Institute of Electrical and ElectronicsEngineers (IEEE) 1394 “Firewire;” among others. For BIOS flash 362, someof the protocols that I/O adapters 338 may translate include SerialPeripheral Interface (SPI), Microwire, among others. Additionally, theremay be one or more wireless protocol I/O adapters. Examples of wirelessprotocols, among others, are used in personal area networks, such asIEEE 802.15 and Bluetooth, 4.0; wireless local area networks, such asIEEE 802.11-based wireless protocols; and cellular protocols.

In some embodiments, the I/O subsystem 115 is coupled to a TPM control334 to control access to system persistent states, such as secure data,encryption keys, platform configuration information and the like. In oneembodiment, these system persistent states are stored in a TMP NVRAM 173and accessed via NVRAM controller 332

In one embodiment, TPM 334 is a secure micro-controller withcryptographic functionalities. TPM 334 has a number of trust-relatedcapabilities; e.g., a SEAL capability for ensuring that data protectedby a TPM is only available for the same TPM. TPM 334 can protect dataand keys (e.g., secrets) using its encryption capabilities. In oneembodiment, TPM 334 has a unique and secret RSA key, which allows it toauthenticate hardware devices and platforms. For example, TPM 334 canverify that a system seeking access to data stored in computer system300 is the expected system. TPM 334 is also capable of reporting theintegrity of the platform (e.g., computer system 300). This allows anexternal resource (e.g., a server on a network) to determine thetrustworthiness of the platform but does not prevent access to theplatform by the user.

In some embodiments, I/O subsystem 315 also includes a Management Engine(ME) 335, which is a microprocessor that allows a system administratorto monitor, maintain, update, upgrade, and repair computer system 300.In one embodiment, a system administrator can remotely configurecomputer system 300 by editing the contents of the decode table 333through ME 335 via networks 336.

For convenience of explanation, the remainder of the applicationsometimes refers to NVRAM 142 as a PCMS device. A PCMS device includesmulti-layered (vertically stacked) PCM cell arrays that arenon-volatile, have low power consumption, and are modifiable at the bitlevel. As such, the terms NVRAM device and PCMS device may be usedinterchangeably in the following discussion. However it should berealized, as discussed above, that different technologies besides PCMSmay also be utilized for NVRAM 142.

It should be understood that a computer system can utilize NVRAM 142 forsystem memory, mass storage, firmware memory and/or other memory andstorage purposes even if the processor of that computer system does nothave all of the above-described components of processor 310, or has morecomponents than processor 310.

In the particular embodiment shown in FIG. 3, the MSC controller 124 andNVRAM controller 332 are located on the same die or package (referred toas the CPU package) as the processor 310. In other embodiments, the MSCcontroller 124 and/or NVRAM controller 332 may be located off-die oroff-CPU package, coupled to the processor 310 or CPU package over a bussuch as a memory bus (like a DDR bus (e.g., a DDR3, DDR4, etc)), a PCIexpress bus, a desktop management interface (DMI) bus, or any other typeof bus.

Exemplary PCM Bus and Packaging Configurations

FIGS. 4A-M illustrates a variety of different deployments in which theprocessor, near memory and far memory are configured and packaged indifferent ways. In particular, the series of platform memoryconfigurations illustrated in FIGS. 4A-M enable the use of newnon-volatile system memory such as PCM technologies or, morespecifically, PCMS technologies.

While some of the same numerical designations are used across multiplefigures in FIGS. 4A-M, this does not necessarily mean that that thestructures identified by those numerical designations are alwaysidentical. For example, while the same numbers are used to identify anintegrated memory controller (IMC) 331 and CPU 401 in several figures,these components may be implemented differently in different figures.Some of these differences are not highlighted because they are notpertinent to understanding the underlying principles of the invention.

While several different system platform configuration approaches aredescribed below, these approaches fall into two broad categories: splitarchitecture, and unified architecture. Briefly, in the splitarchitecture scheme, a memory side cache (MSC) controller (e.g., locatedin the processor die or on a separate die in the CPU package) interceptsall system memory requests. There are two separate interfaces that “flowdownstream” from that controller that exit the CPU package to couple tothe Near Memory and Far Memory. Each interface is tailored for thespecific type of memory and each memory can be scaled independently interms of performance and capacity.

In the unified architecture scheme a single memory interface exits theprocessor die or CPU package and all memory requests are sent to thisinterface. The MSC controller along with the Near and Far Memorysubsystems are consolidated on this single interface. This memoryinterface must be tailored to meet the memory performance requirementsof the processor and must support a transactional, out-of-order protocolat least because PCMS devices may not process read requests in order. Inaccordance with the above general categories, the following specificplatform configurations may be employed.

The embodiments described below include various types of buses/channels.The terms “bus” and “channel” are used synonymously herein. The numberof memory channels per DIMM socket will depend on the particular CPUpackage used in the computer system (with some CPU packages supporting,for example, three memory channels per socket).

Additionally, in the embodiments described below which use DRAM,virtually any type of DRAM memory channels may be used including, by wayof example and not limitation, DDR channels (e.g., DDR3, DDR4, DDR5,etc). Thus, while DDR is advantageous because of its wide acceptance inthe industry, resulting price point, etc., the underlying principles ofthe invention are not limited to any particular type of DRAM or volatilememory.

FIG. 4A illustrates one embodiment of a split architecture whichincludes one or more DRAM devices 403-406 operating as near memoryacting as cache for FM (i.e., MSC) in the CPU package 401 (either on theprocessor die or on a separate die) and one or more NVRAM devices suchas PCM memory residing on DIMMs 450-451 acting as far memory. Highbandwidth links 407 on the CPU package 401 interconnect a single ormultiple DRAM devices 403-406 to the processor 310 which hosts theintegrated memory controller (IMC) 331 and MSC controller 124. Althoughillustrated as separate units in FIG. 4A and other figures describedbelow, the MSC controller 124 may be integrated within the memorycontroller 331 in one embodiment.

The DIMMs 450-451 use DDR slots and electrical connections defining aDDR channels 440 with DDR address, data and control lines and voltages(e.g., the DDR3 or DDR4 standard as defined by the Joint ElectronDevices Engineering Council (JEDEC)). The PCM devices on the DIMMs450-451 provide the far memory capacity of this split architecture, withthe DDR channels 440 to the CPU package 401 able to carry both DDR andtransactional protocols. In contrast to DDR protocols in which theprocessor 310 or other logic within the CPU package (e.g., the IMC 331or MSC controller 124) transmits a command and receives an immediateresponse, the transactional protocol used to communicate with PCMdevices allows the CPU 401 to issue a series of transactions, eachidentified by a unique transaction ID. The commands are serviced by aPCM controller on the recipient one of the PCM DIMMs, which sendsresponses back to the CPU package 401, potentially out of order. Theprocessor 310 or other logic within the CPU package 401 identifies eachtransaction response by its transaction ID, which is sent with theresponse. The above configuration allows the system to support bothstandard DDR DRAM-based DIMMs (using DDR protocols over DDR electricalconnections) and PCM-based DIMMs configurations (using transactionalprotocols over the same DDR electrical connections).

FIG. 4B illustrates a split architecture which uses DDR DRAM-based DIMMs452 coupled over DDR channels 440 to form near memory which acts as anMSC. The processor 310 hosts the memory controller 331 and MSCcontroller 124. NVRAM devices such as PCM memory devices reside onPCM-based DIMMs 453 that use DDR slots and electrical connections onadditional DDR channels 442 off the CPU package 401. The PCM-based DIMMs453 provide the far memory capacity of this split architecture, with theDDR channels 442 to the CPU package 401 being based on DDR electricalconnections and able to carry both DDR and transactional protocols. Thisallows the system to be configured with varying numbers of DDR DRAMDIMMs 452 (e.g., DDR4 DIMMS) and PCM DIMMs 453 to achieve the desiredcapacity and/or performance points.

FIG. 4C illustrates a split architecture which hosts the near memory403-406 acting as a memory side cache (MSC) on the CPU package 401(either on the processor die or on a separate die). High bandwidth links407 on the CPU package are used to interconnect a single or multipleDRAM devices 403-406 to the processor 310 which hosts the memorycontroller 331 and the MSC controller 124, as defined by the splitarchitecture. NVRAM such as PCM memory devices reside on PCI Expresscards or risers 455 that use PCI Express electrical connections and PCIExpress protocol or a different transactional protocol over the PCIExpress bus 454. The PCM devices on the PCI Express cards or risers 455provide the far memory capacity of this split architecture.

FIG. 4D is a split architecture which uses DDR DRAM-based DIMMs 452 andDDR channels 440 to form the near memory which acts as an MSC. Theprocessor 310 hosts the memory controller 331 and MSC controller 124.NVRAM such as PCM memory devices 455 reside on PCI Express cards orrisers that use PCI Express electrical connections and PCI Expressprotocol or a different transactional protocol over the PCI Express link454. The PCM devices on the PCI Express cards or risers 455 provide thefar memory capacity of this split architecture, with the memory channelinterfaces off the CPU package 401 providing multiple DDR channels 440for DDR DRAM DIMMs 452.

FIG. 4E illustrates a unified architecture which hosts both near memoryacting as an MSC and far memory NVRAM such as PCM on PCI Express cardsor risers 456 that use PCI Express electrical connections and PCIExpress protocol or a different transactional protocol over the PCIExpress bus 454. The processor 310 hosts the integrated memorycontroller 331 but, in this unified architecture case, the MSCcontroller 124 resides on the card or riser 456, along with the DRAMnear memory and NVRAM far memory.

FIG. 4F illustrates a unified architecture which hosts both the nearmemory acting as an MSC and the far memory NVRAM such as PCM, on DIMMs458 using DDR channels 457. The near memory in this unified architecturecomprises DRAM on each DIMM 458, acting as the memory side cache to thePCM devices on that same DIMM 458, that form the far memory of thatparticular DIMM. The MSC controller 124 resides on each DIMM 458, alongwith the near and far memory. In this embodiment, multiple memorychannels of a DDR bus 457 are provided off the CPU package. The DDR bus457 of this embodiment implements a transactional protocol over DDRelectrical connections.

FIG. 4G illustrates a hybrid split architecture, whereby the MSCcontroller 124 resides on the processor 310 and both near memory and farmemory interfaces share the same DDR bus 410. This configuration usesDRAM-based DDR DIMMs 411 a as near memory acting as an MSC with thePCM-Based DIMMs 411 b (i.e., far memory) residing on the same memorychannel of the DDR bus 410, using DDR slots and NVRAM (such as PCMmemory devices). The memory channels of this embodiment carry both DDRand transactional protocols simultaneously to address the near memoryand far memory DIMMs, 411 a and 411 b, respectively.

FIG. 4H illustrates a unified architecture in which the near memory 461a acting as a memory side cache resides on a mezzanine or riser 461, inthe form of DRAM-based DDR DIMMs. The memory side cache (MSC) controller124 is located in the risers DDR and PCM controller 460 which may havetwo or more memory channels connecting to DDR DIMM channels 470 on themezzanine/riser 461 and interconnecting to the CPU over high performanceinterconnect(s) 462 such as a differential memory link. The associatedfar memory 461 b sits on the same mezzanine/riser 461 and is formed byDIMMs that use DDR channels 470 and are populated with NVRAM (such asPCM devices).

FIG. 4I illustrates a unified architecture that can be used as memorycapacity expansion to a DDR memory subsystem and DIMMs 464 connected tothe CPU package 401 on its DDR memory subsystem, over a DDR bus 471. Forthe additional NVM-based capacity in this configuration, the near memoryacting as a MSC resides on a mezzanine or riser 463, in the form of DRAMbased DDR DIMMs 463 a. The MSC controller 124 is located in the riser'sDDR and PCM controller 460 which may have two or more memory channelsconnecting to DDR DIMM channels 470 on the mezzanine/riser andinterconnecting to the CPU over high performance interconnect(s) 462such as a differential memory link. The associated far memory 463 b sitson the same mezzanine/riser 463 and is formed by DIMMs 463 b that useDDR channels 470 and are populated with NVRAM (such as PCM devices).

FIG. 4J is a unified architecture in which a near memory acting as amemory side cache (MSC) resides on each and every DIMM 465, in the formof DRAM. The DIMMs 465 are on a high performance interconnect/channel(s)462, such as a differential memory link, coupling the CPU package 401with the MSC controller 124 located on the DIMMs. The associated farmemory sits on the same DIMMs 465 and is formed by NVRAM (such as PCMdevices).

FIG. 4K illustrates a unified architecture in which the near memoryacting as a MSC resides on every DIMM 466, in the form of DRAM. TheDIMMs are on high performance interconnect(s) 470 connecting to the CPUpackage 401 with the MSC controller 124 located on the DIMMs. Theassociated far memory sits on the same DIMM 466 and is formed by NVRAM(such as PCM devices).

FIG. 4L illustrates a split architecture which uses DDR DRAM-based DIMMs464 on a DDR bus 471 to form the necessary near memory which acts as aMSC. The processor 310 hosts the integrated memory controller 331 andmemory side cache controller 124. NVRAM such as PCM memory forms the farmemory which resides on cards or risers 467 that use high performanceinterconnects 468 communicating to the CPU package 401 using atransactional protocol. The cards or risers 467 hosting the far memoryhost a single buffer/controller that can control multiple PCM-basedmemories or multiple PCM-based DIMMs connected on that riser.

FIG. 4M illustrates a unified architecture which may use DRAM on a cardor riser 469 to form the necessary near memory which acts as a MSC.NVRAM such as PCM memory devices form the far memory which also resideson the cards or risers 469 that use high performance interconnects 468to the CPU package 401. The cards or risers 469 hosting the far memoryhosts a single buffer/controller that can control multiple PCM-baseddevices or multiple PCM based DIMMs on that riser 469 and alsointegrates the memory side cache controller 124.

In some of the embodiments described above, such as that illustrated inFIG. 4G, the DRAM DIMMS 411 a and PCM-based DIMMS 411 b reside on thesame memory channel. Consequently the same set of address/control anddata lines are used to connect the CPU to both the DRAM and PCMmemories. In order to reduce the amount of data traffic through the CPUmesh interconnect, in one embodiment, a DDR DIMM on a common memorychannel with a PCM-based DIMM is configured to act as the sole MSC fordata stored in the PCM-based DIMM. In such a configuration, the farmemory data stored in the PCM-based DIMM is only cached in the DDR DIMMnear memory within the same memory channel, thereby localizing memorytransactions to that particular memory channel.

Additionally, to implement the above embodiment, the system addressspace may be logically subdivided between the different memory channels.For example, if there are four memory channels, then ¼ of the systemaddress space may be allocated to each memory channel. If each memorychannel is provided with one PCMS-based DIMM and one DDR DIMM, the DDRDIMM may be configured to act as the MSC for that ¼ portion of thesystem address space.

The choice of system memory and mass storage devices may depend on thetype of electronic platforms on which embodiments of the invention areemployed. For example, in a personal computer, tablet computer, notebookcomputer, smartphone, mobile phone, feature phone, personal digitalassistant (PDA), portable media player, portable gaming device, gamingconsole, digital camera, switch, hub, router, set-top box, digital videorecorder, or other devices that have relatively small mass storagerequirements, the mass storage may be implemented using NVRAM massstorage 152A alone, or using NVRAM mass storage 152A in combination witha flash/magnetic/optical mass storage 152B. In other electronicplatforms that have relatively large mass storage requirements (e.g.,large-scale servers), the mass storage may be implemented using magneticstorage (e.g., hard drives) or any combination of magnetic storage,optical storage, holographic storage, mass-storage flash memory, andNVRAM mass storage 152A. In such a case, system hardware and/or softwareresponsible for storage may implement various intelligent persistentstorage allocation techniques to allocate blocks of persistent programcode and data between the FM 151B/NVRAM storage 152A and aflash/magnetic/optical mass storage 152B in an efficient or otherwiseuseful manner.

For example, in one embodiment a high powered server is configured witha near memory (e.g., DRAM), a PCMS device, and a magnetic mass storagedevice for large amounts of persistent storage. In one embodiment, anotebook computer is configured with a near memory and a PCMS devicewhich performs the role of both a far memory and a mass storage device(i.e., which is logically partitioned to perform these roles as shown inFIG. 3). One embodiment of a home or office desktop computer isconfigured similarly to a notebook computer, but may also include one ormore magnetic storage devices to provide large amounts of persistentstorage capabilities.

One embodiment of a tablet computer or cellular telephony device isconfigured with PCMS memory but potentially no near memory and noadditional mass storage (for cost/power savings). However, thetablet/telephone may be configured with a removable mass storage devicesuch as a flash or PCMS memory stick.

Various other types of devices may be configured as described above. Forexample, portable media players and/or personal digital assistants(PDAs) may be configured in a manner similar to tablets/telephonesdescribed above, gaming consoles may be configured in a similar mannerto desktops or laptops. Other devices which may be similarly configuredinclude digital cameras, routers, set-top boxes, digital videorecorders, televisions, and automobiles.

Embodiments of a MSC Architecture

In one embodiment of the invention, the bulk of DRAM in system memory isreplaced with PCM. As previously discussed, PCM provides significantimprovements in memory capacity at a significantly lower cost relativeto DRAM and is non-volatile. However, certain PCM characteristics suchas asymmetrical Read-vs-Write performance, write cycling endurancelimits, as well as its non-volatile nature makes it challenging todirectly replace DRAM without incurring major software changes. Theembodiments of the invention described below provide asoftware-transparent way to integrate PCM while also enabling newerusages through software enhancements. These embodiments promote asuccessful transition in memory subsystem architecture and provide a wayto consolidate both memory and storage using a single PCM pool, thusmitigating the need for a separate non-volatile storage tier in theplatform.

The particular embodiment illustrated in FIG. 5A includes one or moreprocessor cores 501 each with an internal memory management unit (MMU)502 for generating memory requests and one or more internal CPU caches503 for storing lines of program code and data according to a specifiedcache management policy. As previously mentioned, the cache managementpolicy may comprise an exclusive cache management policy (in which anyline present in one particular cache level in the hierarchy is notpresent in any other cache level) or an inclusive cache managementpolicy (in which duplicate cache lines are stored at different levels ofthe cache hierarchy). The specific cache management policies which maybe employed for managing the internal caches 503 are well understood bythose of skill in the art and, as such, will not be described here indetail. The underlying principles of the invention are not limited toany particular cache management policy.

Also illustrated in FIG. 5A is a home agent 505 which provides access tothe MSC 510 by generating memory channel addresses (MCAs) for memoryrequests. The home agent 505 is responsible for managing a specifiedmemory address space and resolves memory access conflicts directed tothat memory space. Thus, if any core needs to access a given addressspace, it will send requests to that home agent 505, which will thensend the request to that particular MMU 502. In one embodiment, one homeagent 505 is allocated per MMU 502; however, in some embodiments, asingle home agent 505 may service more than one memory management unit502.

As illustrated in FIG. 5A, a MSC 510 is configured in front of thePCM-based far memory 519. The MSC 510 manages access to a near memory518 and forwards memory access requests (e.g., reads and writes) to thefar memory controller 521 when appropriate (e.g., when the requestscannot be serviced from the near memory 518). The MSC 510 includes acache control unit 512 which operates responsive to a tag cache 511which stores tags which identify the cache lines contained within thenear memory 518. In operation, when the cache control unit 512determines that the memory access request can be serviced from the nearmemory 518 (e.g., in response to a cache hit), it generates a nearmemory address (NMA) to identify data stored within the near memory 518.A near memory control unit 515 interprets the NMA and responsivelygenerates electrical signals to access the near memory 518. Aspreviously mentioned, in one embodiment, the near memory is a dynamicrandom access memory (DRAM). In such a case, the electrical signals mayinclude row address strobe (RAS) and column address strobe (CAS)signals. It should be noted, however, that the underlying principles ofthe invention are not limited to the use of DRAM for near memory.

Another component that ensures software-transparent memory applicationis an optimized PCM far memory controller 521 that manages the PCM farmemory 530 characteristics while still providing the performancerequired. In one embodiment, the PCM controller 521 includes an AddressIndirection Table 520 that translates the MCA generated by the cachecontrol unit 515 to a PDA which is used to directly address the PCM farmemory 530. These translations may occur at the granularity of a “block”which is typically 5 KB. The translation is required as, in oneembodiment, the far memory controller 521 continuously moves the PCMblocks throughout the PCM device address space to ensure no wear-out hotspots due to a high frequency of writes to any specific block. Aspreviously described, such a technique is sometimes referred to hereinas “wear-leveling”.

Thus, the MSC 510 is managed by the cache control unit 512 which allowsthe MSC 510 to absorb, coalesce and filter transactions (e.g., reads andwrites) to the PCM far memory 530. The cache control unit 512 managesall data movement and consistency requirements between the near memory518 and the PCM far memory 530. Additionally, in one embodiment, the MSCcache controller 512 interfaces to the CPU(s) and provides the standardsynchronous load/store interface used in traditional DRAM based memorysubsystems.

Exemplary read and write operations will now be described within thecontext of the architecture shown in FIG. 5A. In one embodiment, a readoperation will first arrive at the MSC controller 512 which will performa look-up to determine if the requested data is present (e.g., utilizingthe tag cache 511). If present, it will return the data to therequesting CPU, core 501 or I/O device (not shown). If the data is notpresent, the MSC controller 512 will send the request along with thesystem memory address (also referred to herein as the memory channeladdress or MCA) to the PCM far memory controller 521. The PCM controller521 will use the Address Indirection Table 520 to translate the addressto a PDA and direct the read operation to this region of the PCM. Uponreceiving the requested data from the PCM far memory 530, the PCMcontroller 521 will return the requested data to the MSC controller 512which will store the data in the MSC near memory 518 and also send thedata to the requesting CPU core 501, or I/O Device. Subsequent requestsfor this data may be serviced directly from the MSC near memory 518until it is replaced by some other PCM data.

In one embodiment, a memory write operation also first goes to the MSCcontroller 512 which writes it into the MSC near memory 518. In thisembodiment, the data may not be sent directly to the PCM far memory 530when a write operation is received. For example, the data may be sent tothe PCM far memory 530 only when the location in the MSC near memory 518in which the data is stored must be re-used for storing data for adifferent system memory address. When this happens, the MSC controller512 notices that the data is not current in PCM far memory 530 and willthus retrieve it from near memory 518 and send it to the PCM controller521. The PCM controller 521 looks up the PDA for the system memoryaddress and then writes the data to the PCM far memory 530.

In one embodiment, the size of the MSC near memory 518 will be dictatedby the workload memory requirements as well as the near and far memoryperformance. For a DRAM-based MSC, the size may be set to a tenth thesize of the workload memory footprint or the PCM far memory 530 size.Such an MSC is very large compared to conventional caches found incurrent processor/system architectures. By way of example, and notlimitation, for a PCM far memory size of 128 GB, the size of the MSCnear memory can be as large as 16 GB.

FIG. 5B illustrates additional details associated with one embodiment ofthe MSC 510. This embodiment includes a set of logical units responsiblefor commands and addressing including a command buffer tracking unit 542for buffering commands/addresses and a cache access mode check unit 544which selects an MSC operating mode in response to control signal froman MSC Range Register (RR) unit 545. Several exemplary modes ofoperation are described below. Briefly, these may include modes in whichthe near memory is used in a traditional caching role and modes in whichthe near memory 518 forms part of system memory. A tag checking/commandscheduler 550 uses tags from the tag cache 511 to determine whether aparticular cache line is stored in the near memory 518 and a near memorycontroller 515 generates channel address signals (e.g., CAS and RASsignals).

This embodiment also includes a set of logical units responsible fordata routing and processing including a set of data buffers 546 forstoring data fetched from near memory or stored to near memory. In oneembodiment, a prefetch data cache 547 is also included for storing dataprefetched from near memory and/or far memory. However, the prefetchdata cache 547 is optional and is not necessary for complying with theunderlying principles of the invention.

An error correction code (ECC) generator/checker unit 552 generates andchecks ECCs to ensure that data written to or read from near memory isfree from errors. As discussed below, in one embodiment of theinvention, the ECC generator/checker unit 552 is modified to store cachetags. Specific ECCs are well understood by those of ordinary skill inthe art and will therefore not be described here in detail. The channelcontrollers 553 couple the data bus of the near memory 518 to the MSC510 and generate the necessary electrical signaling for accessing thenear memory 518 (e.g., RAS and CAS signaling for a DRAM near memory).

Also illustrated in FIG. 5B is a far memory control interface 548 forcoupling the MSC 510 to far memory. In particular, the far memorycontrol interface 548 generates the MCAs required to address the farmemory and communicates data between the data buffers 546 and farmemory.

As mentioned, the near memory 518 employed in one embodiment is verylarge compared to conventional caches found in current processor/systemarchitectures. Consequently, the tag cache 511 that maintains the systemmemory address translation to near memory addresses may also be verylarge. The cost of storing and looking up the MSC tags can be asignificant impediment to building large caches. As such, in oneembodiment of the invention, this issue is resolved using an innovativescheme that stores the cache tags within the storage allocated in theMSC for ECC protection, thereby essentially removing the cost of storagefor the tags.

This embodiment is illustrated generally in FIG. 5C which shows anintegrated tag cache and ECC unit 554 for storing/managing cache tags,storing ECC data, and performing ECC operations. As illustrated, thestored tags are provided to the tag check/command scheduler 550 uponrequest when performing tag check operations (e.g., to determine if aparticular block of data is stored within the near memory cache 518).

FIG. 5D illustrates the organization of an exemplary set of data 524 anda corresponding ECC 523 and tag 522. As illustrated, the tag 522 isco-located with the ECC 523 in a memory of the tag cache/ECC unit 554(e.g., DDR DRAM in one embodiment). In this example, several blocks ofdata totaling 64 Bytes has been read into the tag cache/ECC unit 554. AnECC check/generator unit 554 a generates an ECC using the data 525 andcompares the generated ECC against the existing ECC 523 associated withthe data. In this example, a 4-Byte ECC is generated for the 64 Bytes ofdata 525. However, the underlying principles of the invention are notlimited to any particular type or size of ECC. Additionally, it shouldbe noted that the term “data” is used broadly herein to refer to bothexecutable program code and data, both of which may be stored in thedata storage 525 shown in FIG. 5D.

In one embodiment, a 3-Byte (24-bit) tag 522 is used with the bitassignments illustrated in FIG. 5D. Specifically, bits 00 to 16 areaddress bits which provide the upper address bits of the cache line. Fora system address having 56 bits (e.g., SPA [55:00]), bits 00 to 16 mapto bits 55-29 of the system address, allowing for the smallest cachesize of 512 MB. Returning to the 3-Byte tag, bits 17-19 are reserved;bits 20-21 are directory bits which provide information on remote CPUcaching of the cache line (e.g., providing an indication as to the otherCPUs on which the line is cached); bits 21-22 indicate the current stateof the cache line (e.g., 00=clean; 01=dirty; 10 and 11=unused); and bit23 indicates whether the cache line is valid (e.g., 1=valid; 0=invalid).

Utilizing a direct-mapped cache architecture as described above, whichallows the near memory address to be directly extracted from the systemmemory address reduces or eliminates the latency cost of looking up thetag store before the MSC 510 can be read, thereby significantlyimproving performance. Moreover, the time to check the cache tags todecide if the MSC 510 has the required data is also eliminated as it isdone in parallel with the ECC check of the data read form the MSC.

Under certain conditions, storing tags with the data may create an issuefor writes. A write first reads the data in order to ensure that it doesnot over-write data for some other address. Such a read before everywrite could become costly. One embodiment of the invention employs adirty line tag cache that maintains the tags of recently-accessed nearmemory addresses (NMAs). Since many writes target recently accessedaddresses, a reasonably small tag cache can get an effective hit rate tofilter most of the reads prior to a write.

Additional details associated with one embodiment of a PCM DIMM 519including a PCM far memory controller 521 and a set of PCM far memorymodules 530 a-i is illustrated in FIG. 5E. In one embodiment, a singlepool of PCM far memory 530 a-i is dynamically shared between systemmemory and storage usages. In this embodiment, the entire PCM pool 530a-i may be subdivided into “blocks” of 4 KB size. A PCM Descriptor Table(PDT) 565 identifies the use of each PCM block as either memory orstorage. For example, each row of the PDT may represent a particularblock with a particular column identifying the use of each block (e.g.,1=memory; 0=storage). In this embodiment, an initial systemconfiguration can partition the PCM blocks within the PCM 530 a-ibetween storage and memory use (i.e., by programming the PDT 565). Inone embodiment, the same table is used to exclude bad blocks and providespare blocks for wearing-leveling operations. In addition, the PDT 565may also include the mapping of each PCMS block to a “logical” blockaddress used by software. In the case of System Memory, the logicalblock address is the same as the MCA or SPA. This association is neededto update the Address Indirection Table (AIT) 563 whenever the PCMSblock is moved due to wear leveling. When this happens the logical blockaddress used by software has to be mapped to a different PCMS DeviceAddress (PDA). In one embodiment, this mapping is stored in the AIT andis updated on every wear-level move.

As illustrated, the PCM controller 521 includes a system physicaladdress (SPA)-to-PCM mapper 556 which operates in response to a wearmanagement unit 555 and an address indirection unit 563 to map SPAs toPCM blocks. In one embodiment, the wear management logic 555 implementsa wear leveling algorithm to account for the fact that the storage cellsof the PCM 530 a-530 i begin to wear out after too many write and/orerase accesses. Wear leveling spreads writes and erases across the PCMdevice's memory cells by, for example, forcing data blocks with lowcycle counts to occasionally move, and thereby allowing high cycled datablocks to be placed in memory cells that stored the low cycled datablocks. Typically, the majority of blocks do not cycle, but high cyclecount blocks are most likely to fail and wear leveling swaps addressesof high cycle count blocks with low cycle count blocks. The wearmanagement logic 555 may track the cycle counts using one or morecounters and registers (e.g., the counters may increment by one eachtime a cycle is detected and the result may be stored in the set ofregisters).

In one embodiment, the address indirection logic 563 includes an addressindirection table (AIT) containing an indication of the PCM blocks towhich write operations should be directed. The AIT may be used toautomatically move blocks between memory and storage usages. From thesoftware perspective, the accesses to all the blocks uses traditionalmemory load/store semantics (i.e., wear leveling and address indirectionoperations occur transparently to software). In one embodiment, AIT isused to translate the SPA that is generated by software to a PDA. Thistranslation is required as the need to uniformly wear the PCMS devices,the data will need to be moved around in PDA space to avoid anyhotspots. When such a move occurs, the relationship between SPA and PDAwill change and the AIT will be updated to reflect this new translation.

Following the SPA to PCM mapping, a scheduler unit 557 schedules theunderlying PCM operations (e.g., reads and/or writes) to the PCM devices530 a-l and a PCM protocol engine 558 generates the electrical signalingrequired for performing the read/write operations. An ECC unit 562performs error detection and correction operations and data buffers 561temporarily buffer data being read from or written to the PCM devices530 a-l. A persistent write buffer 559 is used to hold data that isguaranteed to be written back to PCMS even in the event of an unexpectedpower failure (e.g., it is implemented using non-volatile storage).Flush support logic 560 is included to flush the persistent writebuffers to PCMS, either periodically and/or according to a specifieddata flushing algorithm (e.g., after the persistent write buffers reacha specified threshold).

In one embodiment, the MSC 510 automatically routes storage accessesdirectly to the PCM far memory controller 521 and memory accesses to theMSC cache control unit 512. Storage accesses coming to the PCM farmemory controller 521 are treated as regular reads and writes and theaddress indirection and wear leveling mechanisms described herein areapplied as usual. An additional optimization is employed in oneembodiment of the invention which can be implemented when data needs tomove between storage and memory. Since a common PCM pool 530 a-l isused, data movement can be eliminated or deferred by simply changing thepointers in the translation tables (e.g., the AIT). For example, whendata is transferred from storage to memory, a pointer identifying thedata in a particular physical PCM storage location may be updated toindicate that the same physical PCM storage location is now a memorylocation in system memory. In one embodiment, this is done by hardwarein a software-transparent manner to provide both performance and powerbenefits.

In addition to the software-transparent mode of operation, oneembodiment of the MSC controller 512 provides alternate modes ofoperations as indicated by the MSC range registers (RRs) 545. Thesemodes of operation may include, but are not limited to the following:

1) Direct access of PCM memory for storage class applications. Suchusage will also require the MSC controller 512 to ensure that writessubmitted to PCM 519 are actually committed to a persistent state.

2) Hybrid use of the near memory 518, exposing portions of it tosoftware for direct use while maintaining the remaining as an MSC. Whena portion of near memory 518 is exposed to software for direct use, thatportion is directly addressable within the system address space. Thisallows certain applications to explicitly split their memory allocationbetween a high-performance small region (the near memory 518) and arelatively lower performance bulk region (the far memory 530). Bycontrast, the portion allocated as a cache within the MSC does not formpart of the system address space (but instead acts as a cache for farmemory 530 as described herein).

As previously discussed, the MSC architecture is defined such thatseveral different system partitioning approaches are possible. Theseapproaches fall into two broad buckets:

(1) Split Architecture:

In this scheme the MSC controller 512 is located in the CPU andintercepts al system memory requests. There are two separate interfacesfrom the MSC that exit the CPU to connect to the Near Memory (e.g.,DRAM) and Far memory (e.g., PCM). Each interface is tailored for thespecific type of memory and each memory can be scaled independently interms of performance and capacity.

(2) Unified Architecture:

In this scheme a single memory interface exits the CPU and all memoryrequests are sent to this interface. The MSC controller 512 along withthe Near Memory (e.g., DRAM) and Far Memory (e.g., PCM) subsystem areconsolidated external to the CPU on this single interface. In oneembodiment, this memory interface is tailored to meet the memoryperformance requirements of the CPU and supports a transactional,out-of-order protocol. The Near and Far memory requirements are met in a“unified” manner on each of these interfaces.

Within the scope of the above buckets several different portioningoptions are feasible some of which are described below.

Split Example:

Near Memory: DDR5 DIMM's

Near Memory Interface: One or more DDR5 channelsFar Memory: PCM controller/device on a PCI express (PCIe) card

Far Memory Interface: x16 PCIe, Gen 3

2) Unified Example:

CPU Memory Interface: one or more KTMI (or QPMI) channelsNear/Far Memory with MSC/PCM Controller on a Riser Card

Near Memory Interface off MSC/PCM Controller: DDR5 Interface Far MemoryInterface off MSC/PCM Controller: PCM Device Interface EmbodimentsHaving Different Near Memory Modes of Operation

As discussed above, a two-level memory hierarchy may be used forintroducing fast non-volatile memory such as PCM as system memory whileusing a very large DRAM-based near memory. The near memory may be usedas a hardware-managed cache. However, some applications are not hardwarecache-friendly and, as such, would benefit from alternate ways to usesuch memory. Because there may be several different applications runningon a server at any given time, one embodiment of the invention allowsmultiple usage modes to be enabled concurrently. Additionally, oneembodiment provides the ability to control the allocation of near memoryfor each of these usage modes.

In one embodiment, the MSC controller 512 provides the following modesfor using near memory. As previously mentioned, in one embodiment, thecurrent mode of operation may be specified by operation codes stored inthe MSC range registers (RRs) 545.

(1) Write-Back Caching Mode:

In this mode, all or portions of the near memory 518 is used as a cachefor the PCM memory 530. While in write-back mode, every write operationis directed initially to the near memory 518 (assuming that the cacheline to which the write is directed is present in the cache). Acorresponding write operation is performed to update the PCM far memory530 only when the cache line within the near memory 518 is to bereplaced by another cache line (in contrast to write-through modedescribed below in which each write operation is immediately propagatedto the far memory 530).

In one embodiment, a read operation will first arrive at the MSC cachecontroller 512 which will perform a look-up to determine if therequested data is present in the PCM far memory 518 (e.g., utilizing atag cache 511). If present, it will return the data to the requestingCPU, core 501 or I/O device (not shown in FIG. 5A). If the data is notpresent, the MSC cache controller 512 will send the request along withthe system memory address to the PCM far memory controller 521. The PCMfar memory controller 521 will translate the system memory address to aPCM physical device address (PDA) and direct the read operation to thisregion of the far memory 530. As previously mentioned this translationmay utilize an address indirection table (AIT) 563 which the PCMcontroller 521 uses to translate between system memory addresses and PCMPDAs. In one embodiment, the AIT is updated as part of the wear levelingalgorithm implemented to distribute memory access operations and therebyreduce wear on the PCM FM 530.

Upon receiving the requested data from the PCM FM 530, the PCM FMcontroller 521 returns the requested data to the MSC controller 512which stores the data in the MSC near memory 518 and also sends the datato the requesting processor core 501, or I/O Device (not shown in FIG.5A). Subsequent requests for this data may be serviced directly from thenear memory 518 until it is replaced by some other PCM FM data.

In one embodiment, a memory write operation also first goes to the MSCcontroller 512 which writes it into the MSC near memory acting as a FMcache 518. In this embodiment, the data may not be sent directly to thePCM FM 530 when a write operation is received. For example, the data maybe sent to the PCM FM 530 only when the location in the MSC near memoryacting as a FM cache 518 in which the data is stored must be re-used forstoring data for a different system memory address. When this happens,the MSC controller 512 notices that the data is not current in the PCMFM 530 and will thus retrieve it from near memory acting as a FM cache518 and send it to the PCM FM controller 521. The PCM controller 521looks up the PDA for the system memory address and then writes the datato the PCM FM 530.

(2) Near Memory Bypass Mode:

In this mode all reads and writes bypass the NM acting as a FM cache 518and go directly to the PCM far memory 530. Such a mode may be used, forexample, when an application is not cache friendly or requires data tobe committed to persistence at the granularity of a cache line. In oneembodiment, the caching performed by the processor caches 503 and the NMacting as a FM cache 518 operate independently of one another.Consequently, data may be cached in the NM acting as a FM cache 518which is not cached in the processor caches 503 (and which, in somecases, may not be permitted to be cached in the processor caches 503)and vice versa. Thus, certain data which may be designated as“uncacheable” in the processor caches 503 may be cached within the NMacting as a FM cache 518.

(3) Near Memory Read-Cache Write Bypass Mode:

This is a variation of the above mode where read caching of thepersistent data from PCM 519 is allowed (i.e., the persistent data iscached in the MSC 510 for read-only operations). This is useful whenmost of the persistent data is “Read-Only” and the application usage iscache-friendly.

(5) Near Memory Read-Cache Write-Through Mode:

This is a variation of the previous mode, where in addition to readcaching, write-hits are also cached. Every write to the MSC near memory518 causes a write to the PCM far memory 530. Thus, due to thewrite-through nature of the cache, cache-line persistence is stillguaranteed.

(5) Near Memory Direct Access Mode:

In this mode, all or portions of the near memory are directly visible tosoftware and form part of the system memory address space. Such memorymay be completely under software control. Any data movement from the PCMmemory 519 to this region of near memory requires explicit softwarecopies. Such a scheme may create a non-uniform memory address (NUMA)memory domain for software where it gets much higher performance fromnear memory 518 relative to PCM far memory 530. Such a usage may beemployed for certain high performance computing (HPC) and graphicsapplications which require very fast access to certain data structures.This near memory direct access mode is equivalent to “pinning” certaincache lines in near memory. Such pinning may be done effectively inlarger, multi-way, set-associative caches.

Table A below summarizes each of the above-described modes of operation.

TABLE A Mode Reads Writes Write-Back Cache Allocate on Miss Allocate onMiss Writeback on Dirty Evict Writeback on Dirty Evict Cache BypassBypass to Far Memory Bypass to Far Memory Read Cache/Write Allocate onMiss Bypass to Far Memory Bypass Invalidate Cached Line Read Cache/WriteAllocate on Miss Update only on Hit Through Write through to Far MemoryDirect Access Read Direct from Near Write Direct to Near Memory MemoryNo Far Memory Access No Far Memory Access

The processor and chipset components used to implement the above modesof operation include the following:

(1) A Memory-Side-Cache Controller 512 that manages the near memory in atwo layer memory (2LM) hierarchy.

(2) A set of Range Registers 545 (see FIG. 5B) in the memory-side-cache510 that determines the system address ranges for each of theabove-described operating modes.

(3) A mechanism to acknowledge write completions from the PCM memorysubsystem 519 to the MSC controller 515.

(5) A mechanism to invalidate lines in the near memory 518.

(5) A flush engine to evict dirty lines to PCM and invalidate inspecified regions of the near memory address space.

In one embodiment, the memory ranges for each of the usage modes arecontiguous in system address space. However multiple, disjoint, regionsmay use the same mode. In one embodiment, each mode range registerwithin the set of MSC RRs 545 provides the following information:

(1) the mode of operation (e.g., write-back, near memory bypass mode,etc);

(2) the range base in the system address space (e.g., at 2 MBgranularity or greater); and

(3) a range mask field which identifies the size of the region.

In one embodiment, the number of modes supported isimplementation-specific but it is assumed that only one contiguoussystem address range is available for each mode of operation. If a nearmemory direct access range register is specified, then it is assumedthat this is will be mapped to a contiguous region starting at thebottom of the near memory address space. Such a contiguous region mustbe smaller than the size of near memory. Additionally, if any of thecaching modes are being used, the direct access region size must besmaller than the near memory size to allow for adequate cache size forthe required performance. Such allocation of near memory for variousmodes may be configurable by the user.

In summary, one embodiment of the invention is implemented in accordancewith the following set of operations:

(1) When any Read or Write Access reaches the Memory-Side-Cachecontroller 512, it checks the Range Registers 545 (FIG. 5B) to determinethe current mode of operation.

(2) For any read cache/write bypass access, the MSC controller 512checks to see if the address is currently cached. If it is, it mustinvalidate the line before sending the write completion back to thesource.

(3) For any Write Bypass direct PCM operation, the MSC Controller 512awaits a completion back from the PCM controller 521 to ensure that thewrite is committed to a globally visible buffer.

(4) Any Read or Write to the Direct Access mode space in Near Memory, isdirected to the appropriate region of Near Memory. No transactions aresent to the PCM memory.

(5) Any change in the Range Register configuration to increase ordecrease any existing region or add a new region, will require flushingof appropriate cached regions to PCM. For example, if software wishes toincrease the size of the Direct Access mode region by reducing theWrite-Back Cache region, it may do so by first evicting and invalidatingthe appropriate portion of the Near Memory Region and then changing theNear Memory Direct Access Mode Range Register. The MSC Controller 510will then know that future caching is done to a smaller Near MemoryAddress Space.

One particular embodiment of the invention in which the system physicaladdress (SPA) space is divided up among multiple MSCs is illustrated inFIG. 6A. In the illustrated embodiment, MSC cache 654 and controller 656are associated with SPA region 667 a; MSC cache 655 and controller 657are associated with SPA region 667 b; MSC cache 661 and controller 663are associated with SPA region 667 c; and MSC cache 660 and controller662 are associated with SPA region 667 d. Two CPUs, 670 and 671, areillustrated, each with four cores, 650 and 651, respectively, and a homeagent, 652 and 653, respectively. The two CPUs, 670 and 671, are coupledto a common far memory controller 666 via far memory interfaces, 659 and665, respectively.

Thus, in FIG. 6A, the entire SPA memory space is subdivided intoregions, with each region being associated with a particular MSC andcontroller. In this embodiment, a given MSC may have a non-contiguousSPA space allocation but no two MSCs have will have overlapping SPAspace. Moreover, the MSCs are associated with non-overlapping SPA spaceand no inter-MSC coherency techniques are required.

Any of the near memory modes described above may be employed on thearchitecture shown in FIG. 6A. For example, each MSC controller 656-657,662-663 may be configured to operate in Write-Back Caching Mode, NearMemory Bypass Mode, Near Memory Read-Cache Write Bypass Mode, NearMemory Read-Cache Write-Through Mode, or Near Memory Direct Access Mode.As previously discussed, the particular mode is specified within therange register (RR) 655 for each MSC 610.

In one embodiment, different MSCs may concurrently implement differentmodes of operation. For example, the range registers of MSC controller656 may specify the Near Memory Direct Access mode, the range registersof MSC controller 657 may specify the Write Back Cache mode, the rangeregisters of MSC controller 662 may specify the Read Cache/Write Bypassmode, and MSC controller 663 may specify the Read Cache/Write Throughmode. In addition, in some embodiments, individual MSCs may concurrentlyimplement different modes of operation. For example, MSC controller 656may be configured to implement near memory direct access mode forcertain system address ranges and a near memory bypass mode for othersystem address ranges.

The foregoing combinations are, of course, merely illustrative of themanner in which MSC controllers ma be independently programmed. Theunderlying principles of the invention are not limited to these or anyother combinations.

As described with respect to some of embodiments described above (e.g.,such as that described with respect to FIG. 4G), an MSC and its MSCcontroller are configured to operate on the same memory channel (e.g.,the same physical DDR bus) as the PCM DIMM responsible for thatparticular SPA range. Consequently, in this embodiment, memorytransactions which occur within the designated SPA range are localizedwithin the same memory channel, thereby reducing data traffic throughthe CPU mesh interconnect.

FIG. 6B provides a graphical representation of how the system memoryaddress map 620, near memory address map 621 and PCM address map 622 maybe configured in accordance with embodiments of the invention. Aspreviously discussed, the MSC controller 606 operates in a modeidentified by the range registers (RRs) 605. System memory map 620 has afirst region 602 allocated for near memory direct access mode, a secondregion 603 allocated for near memory bypass mode, and a third region 605allocated for write-back cache mode. The MSC controller 606 providesaccess to the near memory as indicated by near memory address map 621,which includes a first region 608 allocated to a Write Back Cache mode,and a second region 609 allocated to a Near Memory Direct Access Mode.As illustrated, Near Memory Cache Bypass operations are provideddirectly to the PCM controller 610 operating in accordance with the PCMaddress map 622, which includes a Near Memory Bypass region 611 (forNear Memory Bypass mode) and a Write-Back Cache region 612 (forWrite-Back Cache mode). Consequently, the system memory map 620, nearmemory address map 621 and PCM address map 622 may be subdivided basedon the specific modes implemented by the MSC controllers.

FIGS. 6C and 6D illustrate addressing techniques employed in oneembodiment of the invention (some of which may have already beengenerally described). In particular, FIG. 6C shows how a system physicaladdress (SPA) 675 maps to a near memory address (NMA) or a PCM deviceaddress (PDA). In particular, the SPA is first decoded by decode logic676 within a processor to identify a home agent 605 (e.g., the homeagent responsible for the decoded address space). Decode logic 677associated with the selected home agent 605 further decodes the SPA 675(or portion thereof) to generate a memory channel address (MCA)identifying an appropriate MSC cache controller 612 allocated to thatparticular SPA space. The selected cache controller 612 then either mapsthe memory access request to a near memory address at 678, followedoptionally by an interleaving operation at 680 (described below) or,alternatively performs an optional interleaving operation at 679,followed by mapping 681 by the PCM far memory controller to a PCM deviceaddress PDA (e.g., using address indirection and wear management asdescribed above).

One embodiment of an optional interleaving process is illustrated inFIG. 6D which shows how software pages can be broken up across multipleMSCs and PCM address spaces using interleaving. In the example shown inFIG. 6D, two pages 682-683 within the SPA space are interleaved bycache-line interleave logic 685 to generate two sets of interleavedlines 685-686 within the MCA space. For example, all of the odd linesfrom the memory pages 682-683 (e.g., lines 1, 3, 5, etc.) may be sent toa first MCA space 685, and all of the even lines from the memory pages682-683 (e.g., lines 2, 5, 6, etc.) may be sent to a second MCA space686. In one embodiment, the pages are 5 KByte pages, although theunderlying principles of the invention are not limited to any page size.PCM controllers 687-688 operating in accordance with Address IndirectionTables (AITs) and wear management logic then rearrange the cache lineswithin the PCM device address (PDA) memory space (as described above).Interleaving of this nature may be used to distribute the workloadacross MSCs 610 and/or PCM devices 619 (e.g., as an alternative tonon-uniform memory address (NUMA)).

Embodiments of the invention may include various steps, which have beendescribed above. The steps may be embodied in machine-executableinstructions which may be used to cause a general-purpose orspecial-purpose processor to perform the steps. Alternatively, thesesteps may be performed by specific hardware components that containhardwired logic for performing the steps, or by any combination ofprogrammed computer components and custom hardware components.

As described herein, instructions may refer to specific configurationsof hardware such as application specific integrated circuits (ASICs)configured to perform certain operations or having a predeterminedfunctionality or software instructions stored in memory embodied in anon-transitory computer readable medium. Thus, the techniques shown inthe figures can be implemented using code and data stored and executedon one or more electronic devices (e.g., an end station, a networkelement, etc.). Such electronic devices store and communicate(internally and/or with other electronic devices over a network) codeand data using computer machine-readable media, such as non-transitorycomputer machine-readable storage media (e.g., magnetic disks; opticaldisks; random access memory; read only memory; flash memory devices;phase-change memory) and transitory computer machine-readablecommunication media (e.g., electrical, optical, acoustical or other formof propagated signals—such as carrier waves, infrared signals, digitalsignals, etc.). In addition, such electronic devices typically include aset of one or more processors coupled to one or more other components,such as one or more storage devices (non-transitory machine-readablestorage media), user input/output devices (e.g., a keyboard, atouchscreen, and/or a display), and network connections. The coupling ofthe set of processors and other components is typically through one ormore busses and bridges (also termed as bus controllers). The storagedevice and signals carrying the network traffic respectively representone or more machine-readable storage media and machine-readablecommunication media. Thus, the storage device of a given electronicdevice typically stores code and/or data for execution on the set of oneor more processors of that electronic device. Of course, one or moreparts of an embodiment of the invention may be implemented usingdifferent combinations of software, firmware, and/or hardware.Throughout this detailed description, for the purposes of explanation,numerous specific details were set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the invention may be practiced without someof these specific details. In certain instances, well known structuresand functions were not described in elaborate detail in order to avoidobscuring the subject matter of the present invention. Accordingly, thescope and spirit of the invention should be judged in terms of theclaims which follow.

1. A computer system comprising: a processor having a plurality of coresfor executing instructions and processing data and one or more processorcaches for caching instructions and data according to a first cachemanagement policy; a first memory channel comprising a first set ofaddress/control and data lines coupled to the processor; a second memorychannel comprising a second set of address/control and data linescoupled to the processor; a first first-level memory and a secondfirst-level memory each having a first set of characteristics associatedtherewith, the first set of characteristics including a first readaccess speed and a first write access speed, the first first-levelmemory coupled to the first memory channel and the second first-levelmemory coupled to the second memory channel; and a first second-levelmemory communicatively coupled to the first memory channel, and a secondsecond-level memory communicatively coupled to the second memorychannel, the first and second second-level memories having a second setof characteristics associated therewith, the second set ofcharacteristics including second read and write access speeds at leastone of which is relatively lower than either the first read access speedor first write access speed, respectively, non-volatility such that thefirst and second second-level memories maintain content if power isremoved, wherein at least a portion of the first first-level memory isconfigured as a cache for instructions and data stored in the firstsecond-level memory and at least a portion of the second first-levelmemory is configured as a cache for instructions and data stored in thesecond second-level memory.
 2. The system as in claim 1 wherein one ofthe first set of characteristics comprises a first power consumptionlevel and the second set of characteristics comprises a second powerconsumption level which is relatively lower than the first powerconsumption level.
 3. The system as in claim 1 wherein one of the firstset of characteristics comprises a first density and the second set ofcharacteristics comprises a second density which is relatively higherthan the first density.
 4. The system as in claim 1 wherein one of thesecond set of characteristics comprises the first and secondsecond-level memories being directly writable so as to not requireerasing of existing data prior to writing.
 5. The system as in claim 1wherein the first and second first-level memories comprises dynamicrandom access memories (DRAMs) and wherein the one or more processorcaches comprise static random access memories (SRAMs).
 6. The system asin claim 5 wherein the first and second second-level memories comprisephase change memories (PCMs).
 7. The system as in claim 6 wherein thePCM memories comprise phase change memory and switch (PCMS) memories. 8.The system as in claim 1 further comprising: a mass storage device forpersistently storing instructions and data, the mass storage devicecommunicatively coupled to the first and second first-level memories andthe first and second second-level memories through an interface.
 9. Thesystem as in claim 1 wherein the first and second first-level memoriesare logically subdivided into a first portion and a second portion, thefirst portion being allocated as system memory and the second portionbeing allocated as a cache for instructions and data stored in the firstand second second-level memories, respectively, according to a secondcache management policy.
 10. The system as in claim 1 wherein the firstwrite access speed is relatively higher than the second write accessspeed but the first read access speed approximates the second readaccess speed.
 11. The system as in claim 10 wherein the first writeaccess speed is at least an order of magnitude higher than the secondwrite access speed.
 12. The system as in claim 1 wherein the first setof characteristics includes a first read access latency and a firstwrite access latency and the second set of characteristics includes asecond read access latency and a second write access latency at leastone of which is relatively higher than either the first read accesslatency or second write access latency, respectively.
 13. The system asin claim 1 wherein the first and second second-level memories arecheaper to manufacture per unit size than the first and secondfirst-level memories.
 14. The system as in claim 1 wherein the firstcache management policy operates independently from the second cachemanagement policy.
 15. The system as in claim 1 wherein the first andsecond memory channels comprise double data rate (DDR) memory channels.16. The system as in claim 15 wherein the first and second first-levelmemories comprise first and second dual in-line memory modules (DIMMs)and the first and second second-level memories comprise third and fourthDIMMs, the first and second DIMMs coupled to individual slots on thefirst memory channel and the second and third DIMMs coupled toindividual slots on the second memory channel.
 17. A computer systemcomprising: a processor having a plurality of cores for executinginstructions and processing data and one or more processor caches forcaching instructions and data according to a first cache managementpolicy; a first memory channel comprising a set of address/control anddata lines coupled to the processor; a first level memory having a firstset of characteristics associated therewith, the first set ofcharacteristics including a first read access speed and a first writeaccess speed, the first level memory communicatively coupled to thefirst memory channel; and a second level memory communicatively coupledto the first memory channel, the second level memory having a second setof characteristics associated therewith, the second set ofcharacteristics including second read and write access speeds at leastone of which is relatively lower than either the first read access speedor first write access speed, respectively, non-volatility such that thesecond level memory maintains its content if power is removed, randomaccess and byte addressability such that instructions or data storedtherein may be randomly accessed at a granularity equivalent to thatused by a memory subsystem of the system.
 18. The system as in claim 17wherein at least a portion of the first level memory is configured as acache for instructions and data stored in the second level memory. 19.The system as in claim 17 wherein one of the first set ofcharacteristics comprises a first power consumption level and the secondset of characteristics comprises a second power consumption level whichis relatively lower than the first power consumption level.
 20. Thesystem as in claim 17 wherein one of the first set of characteristicscomprises a first density and the second set of characteristicscomprises a second density which is relatively higher than the firstdensity.
 21. The system as in claim 17 wherein one of the second set ofcharacteristics comprises the second level memory being directlywritable so as to not require erasing of existing data prior to writing.22. The system as in claim 17 wherein the first level memory comprises adynamic random access memory (DRAM) and wherein the one or moreprocessor caches comprise static random access memories (SRAMs).
 23. Thesystem as in claim 22 wherein the second level memory comprises a phasechange memory (PCM).
 24. The system as in claim 23 wherein the PCMmemory comprises a phase change memory and switch (PCMS).
 25. The systemas in claim 17 further comprising: a mass storage device forpersistently storing instructions and data, the mass storage devicecommunicatively coupled to the first level memory and the second levelmemory through an interface.
 26. The system as in claim 17 wherein thefirst level memory is logically subdivided into a first portion and asecond portion, the first portion being allocated as system memory andthe second portion being allocated as a cache for instructions and datastored in the second level memory according to a second cache managementpolicy.
 27. The system as in claim 17 wherein the first write accessspeed is relatively higher than the second write access speed but thefirst read access speed approximates the second read access speed. 28.The system as in claim 27 wherein the first write access speed is atleast an order of magnitude higher than the second write access speed.29. The system as in claim 17 wherein the first set of characteristicsincludes a first read access latency and a first write access latencyand the second set of characteristics includes a second read accesslatency and a second write access latency at least one of which isrelatively higher than either the first read access latency or secondwrite access latency, respectively.
 30. The system as in claim 17wherein the second level memory is cheaper to manufacture per unit sizethan the first level memory.
 31. (canceled)
 32. (canceled) 33.(canceled)
 34. (canceled)
 35. (canceled)